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(54) Image retrieval method and apparatus 

(57) An image retrieval method and apparatus for 
searching a plurality of images stored in an image file 
for a desired image are disck>&ed. When a designated 
image for designating an image to be retrieved for is in- 
put, and its color is designated, the sizes and colors are 
compared between description information which stores 
the feature of each of regions obtained by dividing each 
of images to be searched stored in the image file, arKi 
feature data of the designated image. In this manner, 
the regions, which are determined to be included in the 
designated image, of image data are obtained, and the 
similarities between the obtained regions of image data 
and the designated image are calculated. Image data 
having higher similarities are read out from the image 
file, and are displayed while being sorted in the de- 
scending order from the highest similarity. 
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Description 

The present invention relates to a content-based Image retrieval method and apparatus for designating an image 
or figure and retrieving image data similar to the designated image or figure In, e.g., an inrage database or an inr\age 
5 file system. 

As an image retrieval method for searching a plurality of images stored in an image file format to find a desired 
Image, the following methods are conventionally used. 

(1) In a soK^alled template matching method, when a designated image as a search template or search key Is 
10 input, the designated Image Is sequentially superposed on image data stored In an image file to be searched, a 

sum total of differences between the pixels of the designated image and those of each image data in the image 
file is calculated, and the minimum value of the sum totals Is determined as a difference between the designated 
Image and the Image data in the image file, thereby selecting Image data having a difference equal to or smaller 
than a predetermined value as similar images. 
15 (2) In another method, image data stored in an inoage file are segmented into some regions having similar pixel 

values, the correspondences between data that describe the Image regions and a designated image or figure are 
calculated using characteristics such as an area, secondary moment, or the like, and innage cteta having the largest 
similarly Is selected as a simitar image. 

20 However, in the conventbnal method (1), the designated Image must be nDoved with respect to Image data to be 

searched, and a difference In the number of corresponding pixels must be calculated in each movement. For this 
reason, the search processing requires much time. Upon search, since an accurate search result cannot be obtained 
unless the designated image Is compared while being moved in units of sufficiently small distances in a two^limensbnal 
region of image data, a very tong calculatbn time Is required. 

25 In the conventkmal method (2), a normal search operation is disturbed when one region of Image data In the image 

file to be searched does not have a one4o-one correspondence with one regbn of an image designated as a search 
key, i.e., when an image in the image file Is over-segmented into smaller regions than the corresponding regbns In a 
designated image. 

In a so-called segmentatkkn technique for automatically segmenting Image data into a plurality of regions, since a 
30 target area which is perceived by a person as one region is often dlviskxially extracted as a plurality of portions, such 
a limitation poses a serious problem In an application. 

When an Image retrieval operatkxi is performed, cak:ulatkxis for checking matching between a designated image 
or figure to be retrieved (to be referred to as a designated innage hereinafter) and each region of all images stored in 
the Image file system are required. When there are a large nunrtber of stored images, the cateulation amount becomes 
35 huge, and the time required for image retrieval also becomes very long. In order to solve this problem, a method for 
decreasing the number of innages to be subjected to nnatchlng calculatkjns by limiting the stored Images using the color 
information of the stored images and the designated image to attain high-speed search processing has already been 
proposed. 

However, in the above-mentioned search method, for example, when a stored image includes a large number of 
40 fine regkxis having various colors, a cokw similar to the cotor of a ck>sed figure designated by the designated image 
is included In the stored image with high possibility indeperniently of an image designated as the designated innage. 
As a result, even when an Image Including such a region is not actually data to be searched, rt Is undesirably selected 
as data for which distance cak:ulations are to be performed, and the effect of the limitation processing cannot be 
expected. 

45 The present invention has been made in consideratk)n of the above-mentbned prior arts, and has as its object to 

provide an image retrieval method and apparatus, which can efficiently retrieve image data even when a search regbn 
in Image data is divisbnalty extracted as a plurality of regk>ns. 

It is still another object of the present invention to provide an innage retrieval method and apparatus, which can 
efficiently retrieve an image by shortening the time required for similarity calculations of images. 

50 It is still another object of the present invention to provide an innage retrieval method and apparatus, which can 

efficiently retrieve an Image by obtaining nnovable regions in which each regbn of an image to be searched and a 
designated image have a specific phase relatbnship therebetween, and discriminating the presence/absence of a 
crossing region of the movable regk>ns. 

It Is still another object of the present Invention to provkle an image retrieve method and apparatus, which can 

55 quickly and accurately search for a desired image in such a manner that edge information of a region of an innage or 
figure pattern or color information of a surrounding region is Input as image description infomnation, the number of 
innage data to be accessed is decreased using various kinds of description intormatbn of image data on the basis of 
the Input image description information, and a limited number of Innages are accessed to retrieve the Image. 
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It is still another object of the present invention to provide an image retrieval method and apparatus, which can 
search for a desired image by inputting only the edge position and Its surrounding cobr of an image. 

It is still another object of the present invention to provide a high^eed image retrieval method and apparatus, 
which allow a reliable operation even when a designated image is a portion of a target image or an image to be retrieved, 
5 and can sufficiently limit the number of images to be tested. 

It is still another object of the present invention to provide an image retrieval method and apparatus, which can 
realize high-speed image search processing by utilizing information associated with the area of a closed region of a 
designated image to be retrieved for. 

It is still another object of the present invention to provide an image retrieval method and apparatus, which can 
10 realize high-speed image search processing by limiting the number of irr^ges to be tested and, thereafter, discriminating 
the similarities of only the limited images. 

Other features and advantages of the present invention will be apparent from the following description taken in 
conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts 
throughout the figures thereof. 

IS The accompanying drawings, which are incorporated in and constitute a part of the specif icatk>n, illustrate embod- 

iments of the invention and, together with the description, serve to explain the principle of the invent bn by way of 
example only. 

Fig. 1 is a block diagram showing the hardware arrangement of an image retrieval apparatus according to an 
20 embodiment of the present invention; 

Fig. 2 is a functional bkxk diagram showing the functional arrangement of the image retrieval apparatus according 
to the embodiment of the present invention; 

Fig. 3 is a view showing an example of the inr^ge retrieval apparatus of the embodiment shown in Fig. 1 ; 
Fig. 4 is a functional bkx;k diagram showing the arrangement of a single region processing unit in the first embod- 
25 iment of the present inventbn; 

Fig. 5 is a view illustrating an example of a designated image m the first embodiment; 

Fig. 6 is a view illustrating an example of the segmentation result of image data to be tested; 

Fig. 7 is a flow chart showing the bourtdary line determination processing in a movable regkxi calculation sectkxi 

in the first embodiment; 

30 Fig. 8 is a view for explaining the boundary line detected in the boundary line determination processing shown in 

Fig- 7; 

Fig. 9 is a view for explaining an example of a movable region as a single region in a geometric parameter space; 
Fig. 10 is a flow chart showing the nnovable region discrimination processing in a region information discrimination 
section in the first errtbodiment; 
35 Fig. 11 is a f tow chart showing the processing for discriminating the presence/absence of an intemal crossing point 

in the movable region discrimination processing shown in Fig. 10; 
Fig. 12 is a view illustrating a compound region inserted in a single region memory; 

Fig. 13 is a flow chart showing the flow of generation and processing of a compound region in a compound region 
processing unit; 

40 Fig. 14 is a functional block diagram showing the arrangement of the region informatk>n discriminatton sectton for 

individual compound regions in the compound region processing unit of the first embodiment; 

Fig. 15 is a view illustrating an example of a compound regkxi in the first embodiment; 

Fig. 16 is a view showing the boundary line of a region detected in the boundary line detection processing; 

Fig. 17 is a view showing an example of a movable region as a single region in the geometric parameter space; 
45 Fig. 1 8 is a view showing an example of a movable region as a compound region in the geometric parameter space; 

Fig. 19 is a view illustrating an example of a compound regon obtained by the first embodiment; 

Fig. 20 is a flow chart showing the processing for discriminating the presence/absence of an internal crossing point 

in the second embodiment; 

Fig. 21 is a functtonal block diagram showing the functional arrangement of a single regkxi processing unit in the 
50 third embodiment; 

Fig. 22 is a bkx:k diagram showing the arrangement of an interactive image processing apparatus comprising an 
image retrieval function according to the fourth ennbodiment; 

Fig. 23 is a schematic functional bkx:k diagram showing the functional arrangement of image retrieval processing 
of the fourth embodiment; 
SB Fig. 24 is a view showing an example of a generated figure pattem (designated image); 

Fig. 25 is a flow chart showing the details of the edge information acquisition processing; 

Fig. 26 is a flow chart showing the details of the corresponding region determination processing; 

Fig. 27 is a flow chart showing the details of the designated edge presence/absence discriminatton processing; 
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Fig. 28 is a flow chart showing the details of the surrounding region determination processing; 

Fig. 29 is a flow chart showing the details of the total similarity determination processing; 

Fig. 30 is a flow chart showing another embodiment of the designated edge presence/absence discrimination 

processing; 

5 Fig. 31 is a block diagram showing the arrangement of an image database system according to the fifth embodiment 

of the present invention; 

Fig. 32 is a flow chart showing the image retrieval processing sequence in an image database; 
Fig. 33 is a view showing an example of a target inr^age that a searcher wants to locate; 

Fig. 34 is a view showing an example of an edge line presented by the searcher to retrieve the target image shown 
10 in Fig. 33; 

Fig. 35 is a view showing the result of iniage dilation processing for the edge line presented in Fig. 34; 

Fig. 36 is a view showing an example of an edge surrounding region generated based on the edge line shown in 

Fig. 34; 

Figs. 37A to 371 are views for explaining the flow of the image processing for obtaining the distance values from 
IS the edge line on the basis of the edge line and its surrounding region; 

Fig. 38 is a view for explaining the matching method between an input edge line and a candidate edge line; 
Fig. 39 is a view showing an example of a curve which is input as a portion corresponding to the edges of an 
original image; 

Fig. 40 is a view showing an example of approximation of the curve edge by a polygonal line; 
20 Fig. 41 is a schematic view showing the image retrieval method according to the sixth embodiment; 

Figs. 42 A to 42C are views showing the formats of blocks in the sixth embodiment; 

Fig. 43 is a graph showing an example of a histogram used for calculating cotor feature vectors; 

Fig. 44 is a flow chart showing the operation of an irtdex generation unit; 

Fig. 45 is a view showing an exannple of an index table in units of resolutions; 
25 Fig. 46 is a flow chart showing the operation of a table search unit; 

Fig. 47 is a flow chart showing the bkx;k size determination processing; 

Figs. 48A and 48B are views showing the format of a block according to a modificatran of the sixth embodiment; 
Fig. 49 is a bkx:k diagram showing the functk)nal arrangement of an image retrieval apparatus according to the 
seventh embodiment of the present inventkMi; 
30 Figs. 50A to 50C are views for explaining an image search operation in the seventh embodiment, in which Fig. 

50 A is a view showing a stored image as a search target wanted by a searcher. Fig. SOB is a view showing an 
example of near-equal-cotor regions obtained by segmenting the stored image, and Fig. 50C is a view showing 
an example of a designated image generated by the searcher; 

Fig. 51 is a flow chart showing the processing upon registration of an image in an image database system in the 
35 image retrieval apparatus of the seventh embodiment; 

Fig. 52 is a flow chart showing the image search processing in the image retrieval apparatus of the seventh embod- 
iment; 

Fig. 53 is a view for explaining ccNmparison between the cotors of respective regkxis in the stored image and the 
color of the designated image in the seventh embodiment; and 
40 Figs. 54A and 548 are views for explaining the eighth embodiment of the present invention, in which Fig. 54A is 

a view showing the state wherein an HSV space is divided into bkx:ks. and Fig. 54B is a view for explaining the 
processing for searching for a color similar to that of a designated image in units of blocks. 

The preferred embodiments of the present invention will be described in detail below with reference to the accom- 
45 panying drawings. 

[First Embodiment) 

Fig. 1 is a block diagram showing the arrangement of an interactive image processing apparatus comprising an 

50 image retrieval function as a typical embodinoent of the present inventkjn. 

The image processing apparatus comprises a CPU 1 for executing image retrieval processing, a ROM 2 for storing 
a control program and a processing program to be executed by the CPU 1 , a F=tAM 3 used as work areas of the control 
program and the processing program, a file system 4 comprising a magnetic disk, a magnetooptical disk, or the like, 
which stores image files 109 for storing iniage data to be searched, and description files 110 for describing the features 

55 of image data in the image files 1 09 to be tested, a retrieval processing menr>ory 5 as a RAM used for retrieval processing 
(to be described later), a display (DP) 111 for displaying retrieved images or displaying messages required for the 
search processing, a keyboard (KB) 7 used for inputting various commands to perform a retrieval operation, a mouse 
8 used as an auxiliary input device for the retrieval operatbn, a printer 9 for outputting retrieved results, a similarity 
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calcutation processor 10 for calculating the similarity between an input image partem and image data, a disk controller 
6 for controlling a disk drive unit 12, and a CPU bus 11 for Interconnecting the respective constituting elements. Note 
that the control program to be executed by the CPU 1 may be stored in, e.g., a hard disk 13 or a floppy disk (FD) 14, 
and may be loaded into the RAM 3 when it is executed. 
5 On the retrieval processing memory 5, a designated image memory 1 01 for temporarily storing a designated Image 

serving as a search key required in the retrieval processing to be described below, a description file buffer 24 for 
temporarily storing the contents of the descriptbn files 110, a single regkxi memory 103, a compound region memory 

105. and a search work area 32 (whk^h will be described in detail later) are assured. Note that the retrieval processing 
memory 5 may be substituted for the RAM 3, the file system 4 may be placed on the hard disk 1 3 or the floppy disk 

10 14, and the similarity calculation processor 10 may be substituted for the CPU1 . 

Fig. 2 Is a block diagram functionally showing the flow of the processing in the image retrieval apparatus according 
to the embodiment of the present invention. 

Referring to Fig. 2, reference numeral 100 denotes an image (to be referred to as a designated image hereinafter) 
for designating an image as a search target from the image files 109. The designated image 100 is stored in the 

15 designated image memory 101 . A single regbn processing unit 102 refers to the description files 110 which describes 
the contents of each image data in the image files 1 09 using the feature of the image data to discriminate if each region 
included in the image data is included in the designated image 100. If the region is included in the image 100, the unit 
102 adds the description information of the region of the image data and information of a movable region of the des- 
ignated image 100 to the single region memory 103. Reference numeral 104 denotes a compound region processing 

20 unit for discriminating if a conditbn that the designated image 100 simultaneously contains alt regk)ns of image data 
constituting a compound regk>n is satisfied. If the image 100 simultaneously contains all the regions, the unit 104 adds 
the descriptk>n information of the regions and information of a movable regbn of the designated image 100 to the 
compound region memory 105. 

Reference numeral 106 denotes a maximum value selection unit for cak:ulating the similarities between the com- 

25 pound regions stored in the compourKi regk>n memory 105 by the compound region processing unit 104, and the 
designated image 1 00. and selecting a maximum value of the calculated similarity values as a similarity for each image 
datum. Reference numeral 107 denotes a sorting unit for receiving the similarities of the designated image 100 with 
respect to the respective image data, and selecting K image data in the descending order from the largest similarity 
value (highest similarity). Reference numeral 108 denotes an image data access unit for accessing the image files 109 

30 to read out the K image data selected by the sorting unit 1 07, and delaying the readout image data on the display 111. 

The single regkjn processing unit 102 refers to the designated image memory 101 which stores the designated 
image 100 (or a figure) input by a searcher as a user, and the image descriptbn files 110 which describe the areas, 
boundary point coordinate strings of the individual regions of image data stored in the image files 1 09, and cak:ulates 
a movable regbn as a region in which the designated image 100 is movable under the condition that each region 

35 described in the image description file 110 is included in the designated image 100. thus discriminating if a conditkxi 
that the region is included in the designated image 100 is satisfied. If the region is included in the designated image 
100, the unit 102 adds information indk^ting the movable region of the designated image 100 to the single region 
memory 103. 

The compound regbn processing unit 104 receives the contents (designated image) of the designated image 
40 memory 101, the regk>n information in the single regkxi rrtemory 103, and the compound region information in the 
compound region memory 105, and generates a compound regkxi as a combination of each single region, and a 
compound regbn as a combinatbn of a compound region and a single region. The unit 104 then discriminates if a 
conditbn that each of these compound regbns is included in the designated image 100 is satisfied. It the region is 
included in the designated image 100, the unit 104 adds the description information of the compound regbn, and 
45 information indicating a movable regbn of the designated image 100 in the compound region to the compound region 
memory 105. 

The maximum value selection unit 106 cateulates the similarities between the designated image 100 and the com- 
pound regions in the compound region memory 105. whbh regbns are selected by the compound region processing 
unit 104 since they satisfy the conditbn that the compound regbn is included in the designated image 100. Then, the 
50 unit 106 selects a maximum value of the similarrties of all the regbns included in one image data, and outputs the 
selected maximum value as the similarity of the image data. 

The sorting unit 107 receives the similarrty values of image data obtained by the maximum value selection unit 

106, and sorts and selects K image data in the descending order from the largest similarity value. The image data 
access unit 108 reads out a plurality of corresponding image data from the image files 109 in the descending order 

55 from the largest similarity value using sorted similarity informatbn of image data stored in an internal memory of the 
sorting unit 107, and displays the readout inr^age data on the display 111 . 

Fig. 3 is a view showing a display example on the screen of the image retrieval apparatus of this embodiment. 
Referring to Fig. 3. reference numeral 201 denotes a figure input window on which a user writes an image (des- 
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ignated image 100) to be retrieved. Reference numeral 202 denotes a condition input window on which the user des- 
ignates, e.g., the type of the target innage. Reference numeral 203 denotes a retrieval result display window for dis- 
playing images as retrieval results in the order of, e.g. , similarity values. Reference numeral 204 denotes a color palette, 
which is used by the user to select cotor information to be assigned to the designated image. Reference numeral 205 
s denotes an additional information display window for displaying additional information added to images as the retrieval 
results. 

In the above-mentioned arrangement, the user designates and inputs, e.g., the type of image data to be retrieved 
in the condition input window 202, inputs a figure which roughly depicts a partial image present in the Image data to 
be retrieved in the figure input window 201, and designates the color of the figure using the color palette 204. 
10 As a result of retrieving image data on the basis of the designated image search conditions, image data corre- 

sponding to the retrieval results are displayed on the retrieval result display window 203 In the order of higher similarities, 
and information added to each image as the retrieval result is displayed on the additional information display window 
205. 

Fig. 4 is a view for explaining the image retrieval processing according to the first embodiment of the present 
15 invention, i.e., a functional block diagram showing the processing for realizing the single region processing unit 102 
shown in Fig. 2. 

Fig. 5 is a view showing an example of the designated image 100. The designated image 1 00 in Fig. 5 is a polygonal 
figure, and the designated image Information is constituted by, e.g., the vertex positions, vertical and horizontal dimen- 
sions, area, color information, and the like of the designated figure. A unit region of the region description informatbn 
20 is obtained by applying the image segmentation technique whk:h is known to those who are skilled in the art to image 
data stored in the image files 109. 

Fig. 6 is a view illustrating an example of the \TX\age segmentation results of image data to be tested. Note that the 
region description information includes the edge point positions, start point, vertical and horizontal dimensbns, area, 
average cotor information, and the like of each region. 
25 In Fig. 4, reference numeral 301 denotes a cotor information discrimination section; 302, a size information dis- 

crimination section; 303, a movable region calculation sectton; and 304, a region informatbn discrimination section. 

The color information discrimination section 301 cabulates the color information difference on the basis of the 
input designated image 1CX) and the regton description information of image data to be searched. If the difference is 
larger than a predetermined threshold value, the section 301 determines a non-coincidence therebetween (i.e., the 
30 region does not become a portton of the designated innage), arKi excludes the region from those to be matched. The 
color information difference can be calculated by the following equation: 

D = IR - rl + IG - gl + IB - bl 

where (R, G, B) is the cok>r of the designated ima^ 100, and (r, g, b) is the color of a region of image data. 

The size information discriminatton section 302 compares the vertical and horizontal dimensions of the designated 

^ image 100 and those of the regions of image data. For example, if the vertical or horizontal dimension dt one of the 
divkied regions shown in Fig. 6 is larger than the vertk:al or horizontal dimension of the designated image 100, the 
sectton 302 determines a non-coincidence therebetween (i.e., the region does not become a portton of the designated 
innage), and excludes the corresponding regton from those to be matched. 

When the color information discrimination section 301 and the size information discrimination section 302 do not 

^ determine a non-coincidence, the movable region cak;ulatton sectton 303 cak:ulates a movable region representing a 
moving range whtoh is included in the designated innage 100 when the designated image 100 Is translated for each 
region of the region description infornnatton of innages to be matched. The regton infonmation discrimination sectton 
304 discriminates if the catoulated movable region is an efnpty region. If the movable region is empty, the sectton 304 
determines a non-coincidence, and excludes the corresponding regton from those to be searched. In this manner, only 
regions which can be included in the designated image 100 are selected as candkJate regions to be matched. 

Fig. 7 is a flow chart showing the determinatton processing of a boundary line as a boundary of a nrK>vable regton 
in the nrtovable region cak;ulatton sectton 303 shown in Fig. 4. 

In step SI , the maximum value is set to be •9999999*, and the equatton of a line segment F is set to be F - aj^ 
+ bj^. The flow then advances to step S2, and from a vertex position Q j (j is an index corresporxJing to each vertex) 

^ of the designated image 1 00, a formula aj»x + bj^ + cj = 0 of an edge line Ej connecting adjacent vertex point coordinate 
posit tons Qj and Qj+1 is obtained. Then, F = aj»Xi + bj«Yi is catoulated using the coefficients (aj, bj) of the formula of 
the edge line Ej and the coordinates (Xi, Yi) of a boundary point Pi for a boundary point string Pi (i is an index corre- 
sponding to each point) of each region of inr^ge data shown in Fig. 6 (step S3). The flow advances to step S4 to 
catoulate a nnaximum value FMAX(j) of F for the boundary point string of one regton (since the vertex string of the 

^ designated image 1 00 must be arranged in the ctockwise order, if the vertex string is arranged in the counterctockwise 
order, the vertex string is re-arranged in the reverse order in advance). If this nnaximum value is larger than the value 
calculated so far, the maximum value is updated in step S5. In step S6, it is checked if processing for all the boundary 
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points P: is completed. If NO in step S6, the flow advances to step S7 to increment the index i by 1 , and the flow then 
retums to step S2. 

In this manner, when FMAX(j) which defines the boundary line aj«x + bj«y - FMAXQ) - 0 including all the points of 
the region boundary point string Pi is obtained, the FMAX is registered in step S8. and it is checked in step S9 rf the 
above-mentioned processing Is executed for all the edge lines Ej constituted by the vertex string of the designated 
image 100. If NO in step S9, the index j Is incremented by 1 in step S10, and the flow returns to step S1, thereby 
obtaining all the boundary lines. 

Fig. 8 is a view for explaining the processing result in the flow chart shown in Fig. 7. In Fig. 8, a limit range in which 
a central region 700 in Fig. 8 is included in the designated image 100 (Fig. 5) upon translation of the designated image 
100 is indicated by arrows. The formula of each of lines irxiicated by these arrows is aj«x + bj«y - FMAX(j) - 0. 

The boundary point string PI = (Xi, Yi) of the region 700 satisfies aj*Xi + bj»Yi < FMAXQ) for the region 700 in the 
image shown in Fig. 8. Assuming a geometric parameter space having translation amounts (dx, dy) in the X- and Y- 
directions as axes, from aj»(x - dx) + bj»(y - dy) + cj = 0, the formula corresponding to each edge line is aj»Xi + bj»Yi - 
FMAX(j) ^ aj»(Xi -dx) + bj»(Yi - dy) + cj, and the translation amounts (dx, dy) must satisfy the following condition: 

aj^ + bj^ - (cj + FMAXG)) ^ 0 

From this condition, by drawing boundary lines aj^ + bj^ - (cj + FMAX(j)) = 0 in the (dx, dy) space, the presence 
range of (dx, dy) satisfying the conditions for all the edge lines of the designated image 100 is obtained, as shown in, 
e.g., Fig. 9. If points (dx, dy) that satisfy the conditions for all the edge lines of the designated image 100 are not 
included in a given region of image data, i.e., the region is empty, it is determined that the region of Image data to be 
searched does not fall within the designated image 100. 

Fig. 10 Is a flow chart showing the processing for discriminating If a movable region is an empty region in the region 
Information discrimination section 304 in Fig. 4. 

In step S21 , the positions of crossing points between the boundary lines in the (dx, dy) space are calculated. It is 
checked in step S22 If these crossing points include an intemal crossing point that satisfies the conditbns for all the 
edge lines of the designated image 1 00. If one of the crossing points between the boundary lines satisfies the conditions 
for all the edge lines of the designated image 100, it is determined that the movable region is not empty Note that the 
positbn of each crossing point between the boundary lines is calcuteted by applying the solution of simultaneous 
equations to the formulas of two boundary lines. 

Fig. 1 1 is a fbw chart showing the flow of the processing for discriminating the presence/absence of an internal 
crossing point In step S22 In Fig. 10. 

In step S31 , a crossing point Pi = (Xi, Yi) between boundary lines Is set. It Is checked in turn for all the boundary 
lines If the set crossing point Pi satisfies a condition aj*Xi + bj*Yi ^ cj + FMAX(j) for a boundary line Bj: aj^ + bj*dy - 
(cj'i-FMAX(j)) = 0 (step S33). It is then checked in step S34 if the processing for all the boundary lines is completed. If 
the crossing point satisfies the conditions for all the boundary lines, it Is determined in step S38 that an internal crossing 
point is present, thus ending the processing. If there is a conditkxi which is not satisfied, the above-mentioned dlscrlm- 
inatran is repeated for the next crossing point 

It is checked In step S36 if the discrimination processing for all the crossing points is completed. If the presence 
of an internal crossing point is not determined after the discrimination processing for all the crossing points Is completed, 
the flow advances to step S39 to detenmlne that an intemal crossing point is absent, thus ending the processing. 

As a result of the above discrimination, some regkxis are inserted in the single region memory 103, as shown in, 
e.g.. Fig. 12. 

Fig. 13 Is a flow chart showing the flow of the generation and processing of a compound regk>n in the compound 
region processing unit 104 shown in Fig. 2. In this processing, a compourKt region as a combinatk>n of some pieces 
of regkxi information in the single regkyi memory 1 03 Is generated, and Is Inserted In the compound regkxi memory 105. 

The system sets all the regions in the single regk>n memory 103 as a compound regbn in the compound region 
memory 105 in step S41 . The flow advances to step S42, and a single regk>n i in the single region memory 103 Is 
selected in turn. In step S43, a temporary memory in the search work area 32, whk;h memory temporarily stores a 
compound region generated as a combination of the selected single region I and the compound region (including a 
copied single region) In the compound region memory 105, is cleared. 

The flow advances to step S44. and a compound region j (Including a copied single region) in the compound region 
memory 105 is selected in turn. In step S45, It is checked If the single regbns constituting the selected compound 
region include the selected single region i. If NO In step S45, the flow advances to step S46 to generate a compound 
region k as a combinatk>n of these single regkjns. The flow then advances to step S47 to check if the compound region 
k is included In the designated Image 100. If YES In step S47, the flow advances to step S48, and the compound region 
k is added to the temporary memory. The above-mentioned processing is repeated while incrementing the index j until 
it Is determined In step S49 that the above processing is completed for all the compound regions in the compound 
region memory 1 05. 
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If it is determined in step S49 that the processing for alt the compound regions is completed, the flow advances 
to step S51 , and the compound region In the temporary menrrory Is added to the compound region memory 1 05. There- 
after, the above-nnentioned processing is repeated while repeating the index i in step S53 until It is determined in step 
S52 that the processing for all the regions In the single region menriory 103 Is completed. 
s Fig. 14 is a functional block diagram showing the functional arrangement of the compound region processing unit 

104 for discriminating if each compound region k Is included in the designated Image 100. 

First, a size information dlscriminatk>n section 1301 discriminates the size information of the compound region. 
The area of the compound region k corresponds to the sum of the areas of the single region i and the compound region 
j, and the start point and vertical and horizontal dimensions of the compound region k are determined In such a manner 
10 that the minimum value of the start points of the respective regkxis arKJ the end points calculated based on the start 
points and the vertical and horizontal dimensions is determined as the start point, and the vertical and horizontal di- 
mensions are cak:ulated as the differences from the nnaximum value. 

If the calculated area of the compound region k Is larger than that of the designated image 100 or the vertical or 
horizontal dimenskxi of the compound region k is larger than that of the designated Image 100, it Is determined that 
15 the compound region k Is not Included In the designated inr^age 100, and the compound region k is excluded. 

Then, In order to discriminate If the single region I and the compound region j constituting the compound region k 
are simultaneously present in the designated image 100, a crossing region calculation section 1 302 calculates a cross- 
ing region of movable regions in the geometric parameter space having the translation amounts (dx, dy) of the single 
region i and the compound region j as axes, and a region Information discrimination section 1303 discriminates if the 
20 calculated crossing region is empty. 

As described above, the nriovable regon In the (dx, dy) space is obtained from the fol towing condition: 

aj^ + bj^ - (cj + FMAXfl)) < 0 
Therefore, the crossing region is defined by selecting a smaller one of the values FMAX(j) for the boundary lines of 
the respective regions corresponding to an identcal edge line. 
25 Whether or not a crossing region Is empty Is discriminated in the same manner as in a movable region for a single 

region, i.e., in the movable region discrimination operation and the internal crossing point presence/absence discrim- 
ination operation shown In Figs. 10 and 11. 

Fig. 15 shows an example of the compound regbn. Fig. 16 shows the boundary lines of another region paired with 
Fig. 8, and Fig. 17 shows the nrwvable region in the (dx, dy) space paired with Fig. 9. 
^ With the above-ment toned flow of the compound region generation processing shown In Figs. 13 and 14, Fig. 18 

Is obtained as a region In which the compourKi region shown In Fig. 15 is entirely present In the designated image 100 
shown in Fig. 15. As a result of further dlscrlmlnatton of compourtd regions, a compound region shown In Fig. 19 can 
also be obtained. 

The maximum value selectton unit 106 shown In Fig. 2 catoulates the similarity values of all the compound regtons 
^ detected by the above-mentioned compound regton processing unit 104 from one irr^age, and obtains the maximum 
value of the calculated similarity values. 
The similarity Is catoulated as follows. 

Similarity = 1 - larea of compound region - area of designated Iniagel/area of designated image 
The maximum value selection unit 106 supplies the obtained maximum value as the similarity of the corresponding 
^ iniage to the sorting unit 1 07. The sorting unit 1 07 receives the similarity values for the respective images obtained by 

the maximum value selection unit 106, and sorts and selects K data in the descending order from the largest similarity 

value. 

The image data access unit 108 reads out a plurality of Image data In the descending order from the largest 
similarity value from the image tiles 109 using the sorted Image similarity Information stored in the internal memory of 
^ the sorting unit 107, and displays the readout Image data on the display 111. 

[Second Embodiment] 

The second embodiment can cope with a case wherein the size of a designated Image can be varied. The principle 
^ of the second embodiment will be explained below. 

In the second embodiment, when a designated image Is enlarged to have its origin as the center at an enlargement 
factor "scale" (reductton when scale < 1), all the points P(x, y) on the boundary lines of the designated image move to 
points Q(scale»x, scale«y). If the formula of a straight line passing two points PO and P on a boundary line before the 
enlargement is: 

^ aj»x + bj^ + cj = 0 

for aj = yO - y 
bj 3 x - xO 
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cj = xOy - x«yO then the formula of a straight line passing two points QO and Q on a boundary line of the designated 
image after the enlargement is: 

ak»x + bk«y + ck = 0 

for ak = scale^j 
^ bk - scaleH)j 

ck = scale*scale«cj 

This straight line is parallel to one before the enlargement. Therefore, a point which gives the maximum value 
FMAX for a boundary point string of an identical region remains the same. That is, 

10 FMAX(k) = ak^X + bk*Y = scale»(aj*X + bj»Y) 

= scale*FMAX( j) 

Division of the formula: 

ak-dx + bk-dy - (ck + FMAX(k)) < 0 
for giving an Inclusion condition with "scale" yields: 

aj»dx + bj^ - (scale^j + FMAX) <, 0 
This constitutes one skle of a plane upon assumption of a (dx, dy scale) space. 

Similarly, other edge tines before the enlargement are parallel to those after the enlargement, and a parameter 
region that can exist becomes a polyhedrc^ contained in these planes (in practice, one side is infinity). 

Therefore, by obtaining the crossing regions of the polyhedron, whether or not a plurality of regions can be simul- 
taneously present in the designated image at an identical enlargement factor can be discrimir^ted. 

Fig. 20 is a flow chart showing an embodiment of the compound region processing unit 104 which can realize the 
above-mentioned principle. 

First, a crossing point P(dxi, dyi) is set (S61 ). and aj, bj, and cj are cak:ulated as described above for all edge lines 
j of the designated image in the (dx, dy) space satisfying: 

aj«dx + bfdy - (scale^j + FMAX) < 0 
Then, FMAX is obtained by the operation shown in Fig. 7. In this manner, the formula of the boundary line is obtained 
(S62): 

aj»x + bfy - scale«cj = FMAX 
Next, all the crossing points P = (dxi, dyi) are calculated as crossing points of three non-parallel boundary planes, 
and it is checked if each of the calculated points satisfies the respective conditional formulas. The crossing point of 
the three boundary planes is cak^ulated by solving the following simultaneous equations by a known solutbn of simul- 
taneous equatbns: 

35 ai«dx + bi^ - (scale^i + FMAX(i)) - 0 

aj«dx + bj^ - (scale^j + FMAX(j)) = 0 
ak-dx + bk-dy - (scale^k + FMAX(k)) ^ 0 
where (i, j, k) are three different integers smaller than the number of edge lines of the designated image. 
Whether or not the conditional formulas are satisfied can be discriminated using the following formulas: 
40 (ajKlx + bjnJy - FMAX)/cj <, scale if cj > 0 

(aj^lx + bj^ - FMAX)/cj > scale if cj < 0 
If "scale" can be increased infinitely, a region having an arbitrary shape can be included. Thus, In practice, the 
variable range of "scale" is set to be equal to or smaller than a certain value, e.g. "1 .2", and it is checked if an internal 
crossing point is present within this range. That is, 
^ (ardx + bj-dy - FMAX)/q ^ 1.2 if cj > 0 

(aj^ + bj«dy - FMAX)/q > 1.2 if cj < 0 
With this processing, an appropriate correspondence can be obtained even when the size of the designated image 
varies. 

In this manner, as in Fig. 11, when the conditions are satisfied for all the boundary lines, the flow advances from 
^ step S66 to step S67 to determine the presence of an internal crossing point. Otherwise, the ftow advances from step 
S68 to step S69 to determine the absence of an internal crossing point. 

[Third Embodiment] 

^ The third embodiment of the present invention can cope with a case wherein a designated image is constituted 

by a plurality of regions with different colors. 

Each of regions constituting a designated image has the folbwing information: the vertex string of the designated 
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image, and the vertical and horizontal dinnensions, area, and color information of the region. In this case, the operation 
of the single region processing unit 102 shown in Fig. 4 is nrKx^tied, as will be described below 

Fig. 21 is a block diagram showing the arrangement of the single region processing unit 1 02 in the third embodiment. 

The processing sequence corresponding to Fig. 4 is applied to a combination of region information of each of 
regions In the input designated image and that of each of regions in the image to be matched, and when all the conditions 
are satisfied, the description information of the region and the infomnation of the movable region of the designated 
innage are added to the single region mennory 103. Thereafter, the above-mentioned compound region processing unit 
and the subsequent units are constituted, thereby realizing the search processing based on the designated image 
using a plurality of regions. 

In Fig. 21, the color information discrimination 301, the size information discrimination 302, the movable region 
calculation 303, and the region information discrimination 304 are executed in units of patterns of the designated image, 
as shown in Fig. 4, thus obtaining single region information. 

When a portion of an image is input as a designated image, the designated image is divided into a plurality of 
regions, a plurality of regions with different colors (a single color in each region) are generated by polygonal approxi- 
mation of the boundary lines of the regions, thus generating a plurality of images (figures). As a result, even when a 
portion of an image is input as a designated image, image data including the designated image can be retrieved using 
these image portions. 

As described above, according to this embodiment, since processing for a compound region as a combination of 
a plurality of regions is performed together with processing for a single region, an image to be searched which Is 
divisionally extracted as a plurality of regions can be correctly matched. 

In addition, a movable region in which each region has a specific phase relationship with the designated image is 
calculated, and the presence/absence of a crossing region of the movable regions is discriminated, thus allowing 
efficient discrimination. 

[Fourth Embodiment] 

Fig. 22 is a block diagram showing the arrangement of an Interactive image processing apparatus comprising an 
image retrieval furKtion according to the fourth embodiment of the present invention. The same reference numerals 
in Fig. 22 denote the same parts as in Fig, 1, and a detailed description thereof will be omitted. 

In the retrieval processing memory (SRM) 5, the designated image memory 101 for temporarily storing an image 
serving as a retrieval key required in retrieval processing to be described bebw, the description file buffer 24 for tem- 
porarily storing the contents of the descrtptbn file 110, a corresponding region buffer 26 to be described In detail later, 
a designated edge buffer 28 to be described in detail later, a surrourtding region buffer 30 to be described in detail 
later, and the search work area 32 are assured. 

Fig. 23 is a schematic block diagram showing the image retrieval processing of the fourth embodiment. 

Referring to Fig. 23, reference numeral 121 denotes region designation processing for designating a region rep- 
resenting one portion of a given image; 122, edge designatkjn processing for designating a portkxi of an edge of the 
designated region as an edge which is also present in image data (to be referred to as images to be searched here- 
inafter) in the image files 109 to be matched; and 123, surrounding color deslgnatbn processing for designating sur- 
rounding cotor information of the designated region. The region designation processing 121, the edge designation 
processing 122, and the surrounding color designation processing 123 are executed by a searcher in a man-machine 
interactive manner by operating the keyboard (KB) 7 ami the mouse 8 with reference to an image displayed on the 
display (DP) 111. The designated Image generated by these processing operatbns Is stored In the designated image 
memory 101. 

The above-mentioned processing will be described in detail bebw with reference to Figs. 3 and 24. 

Using the above-mentioned screen shown In Fig. 3, the searcher draws a figure pattem of an Image to be searched 
for on the figure Input window 201 by operating the mouse 8. More specifically, the searcher designates a region serving 
as a partial image of the image to be searched for by a polygon, and designates its color from the color palette 204. 
These operations correspond to the region designation processing 121. Then, the searcher selects and designates 
some of the sides of the designated polygon as edges, which may also be present in images to be searched, by clicking 
the mouse 8. If there is no corresponding edge, the searcher need not perform designation. This operatbn corresponds 
to the edge designation processing 122. Then, the searcher selects the background of the figure using the nrxjuse 8, 
and designates a cotor as the background color of the region from the color palette 204. This operation corresponds 
to the surrounding color designation processing 123. 

Fig. 24 shows an example of a figure pattern (designated image) 1000 generated as described above. 

Referring to Fig. 23, reference numeral 124 denotes corresponding region determinatton processing for cabulating 
the similarity between the designated image 1000 and a candidate region in an image to be searched corresponding 
to a regbn of the designated Image 1000 wrth reference to the features of image data stored In the description files 
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1 1 0, and storing infornnation of the candidate region having a high similarity in the corresponding region buffer 26; 1 25, 
designated edge presence/absence discrimination processing for discriminating the presence/absence of an edge In 
an image to be searched corresponding to the edge of the designated image 1000 with reference to the features of 
image data stored in the description files 110, and storing the discrimination result as designated edge information in 

5 the designated edge buffer 28; and 126, surrounding region determination processing for determining a region In an 
Image to be searched corresponding to the surrounding region of the designated image 1000 with reference to the 
features of image data stored in the description flies 110, and storing information of the surrounding region as sur- 
rounding region information In the surrounding region buffer 30. 

Furthermore, reference numeral 127 denotes total similarity calculation processing for calculating the total slmi- 

10 laritles between the designated Innage 1000 and the respective Images on the basis of the information stored In the 
corresponding region buffer 26, the designated edge buffer 28, and the surrounding region buffer 30; 128, sorting 
processing for receiving the total similarities between the designated Image 1000 and the respective images, and 
selecting a predetermined number of (K) Image data In the descending order from the largest similarity value; and 1 29, 
Image data access processing for accessing the image files 109 using the similarity Information of the images sorted 

IS by the sorting processing 128 to read out K \mage data In the descending order from the largest similarity value, and 
displaying the readout Image data on the display 111 . 

The corresponding region determination processing 124, the designated edge presence/absence discrimination 
processing 125, the surrounding region determination processing 126, and the total similarity calculation processing 
127 will be respectively described in detail below with reference to the flow charts shown In Figs. 25 to 29. 

20 

•Corresponding Region Determination Processing (Figs. 25 and 26) 

In this processing, some regions corresponding to those of the designated image 1000 are selected as candidate 
regions with referencetothe designated image 1000 or figure which Is Input by the searcher and stored in the designated 

25 image memory 101 , the areas, boundary point coordinate strings, and color average values of respective regions of 
image data in the image files 109, and the coordinate strings artd the like of respective edges stored In the description 
files 110, and the description information (stored in the description files 110) of each candidate region and the similarity 
between the candidate region and the designated Image 1000 are output to the corresponding region buffer 26. 
The region description information of Image data obtained from the description files 110 includes the vertical and 

30 horizontal dimensions, area, average color Information, and bark:entric coordinate position of each region, the coordi- 
nate positions of vertices at the two ends of a portion obtained by polygonal approxinnatlon of the edges of the region, 
and edge Information ('1 " If the line segment Is an edge; "0" if the line segment is not an edge) indicating whether or 
not each line segment is an edge. Note that a unit region of the region description infornratlon Is obtained by applying 
known region division processing to an Image. Linear approximatkx) of the edge lines of dIvkJed regkxis Is attained by 

35 known linear approxlmatkin processing (for example, Y Kurozumi, "Polygonal approximation by the minimax method". 
Computer Graphics and Image Processing, Vol 19, pp. 248 - 264 (1982)). 

In order to obtain the edge informatbn indk^atlng whether or not each line segment Is an edge, processing shown 
in Fig. 25 is performed. In this processing. It is checked If a line segment P as one of line segments indk^ting an edge 
of a given region in an image (i) Is an edge, and If the line segment P is an edge, "1 ' Is set; otherwise, "0' Is set. 

40 In step S71 , the line segment P Is dilated by a predetermined number of pixels by known dilatbn processing. This 

line segment will be referred to as an Inflated line segment P' hereinafter. In steps S72 and S73, It Is checked for all 
the pixels of the inflated tine segment P If the pixel of interest Is an edge in an edge image. For this purpose, the edge 
image must be extracted in advance from the Image (I) by edge extractkxi processing. The edge extractkxi processing 
may use a known method. 

45 In this manner, the overlapping ratkD (xp) between the Inflated line segment P' and the edge image Is calculated 

in step S74. If it is determined In step S75 that the ratb (xp) Is equal to or larger than a predetermined threshokJ value 
(dth), it is determined that the line segment P is an edge, and the flow advances to step S76 to set T as the edge 
detection result; if It is determined in step S75 that the ratio (xp) is smaller than the threshold value, it Is determined 
that the line segment P is not an edge, and the flow advances to step S77 to set "O" as the edge detection result. 

so The designated image information stored in the designated image menrwry 1 000 is associated with the designated 

image 1000 (figure) generated as described above with reference to Fig. 3, and includes the vertical and horizontal 
dimensions, area, color infornnation, and barlcentric coordinate posit k>n of the figure, the coordinate pos'rtk>ns of vertices 
at the two ends of a skie of the designated figure, edge information ("1 " if the side Is designated as an edge; "0" if the 
side is not designated as an edge) indk:atlng If each side Is designated as an edge, and color Information of a sur- 

55 rounding region. 

In conskieratk>n of the above description, the corresponding region determination processing 1 24 will be described 
In detail below with reference to the flow chart in Fig. 26. 

In step S81 , a cobr information difference (Dc) Is calculated on the basis of the input designated Image Infornnation 
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and the region description infornnation in the description files 110. If the difference (Dc) is larger than a predetermined 
threshold value, a non -coincidence (the region of interest does not become a corresponding candidate region of the 
designated image 1000) is determined, and the region of interest is excluded from corresponding candidate regions. 
The cobr information difference (Dc) can be calculated based on the following equation: 
5 Dc = IR - rl + IG - gl + IB - bl 

where R, G, and B are the color component values of the designated image, and r, g, and b are the color component 
values of a candidate region. 

In step S82, the vertical and horizontal dimensions of each of the regions which are not excluded in the processing 
in step S81 are compared with those of the designated inr«ge 1000. If the difference between the vertical or horizontal 
dimension of the candidate region of interest and that of the designated image 1000 is larger than a threshold value, 
a non-coincidence is determined, and the region of interest is excluded from the candidate regions. In this comparison, 
a value defined by Ds = IW - wl + IV -vl (w and v are the vertical and horizontal dimensions of the designated image 
1000, and W and V are those of the candidate region) is compared with a predetermined threshold value. 

Furthemnore, in step S83, the bartcentric coordinate position of each of the regions which are not excluded in the 
processing in step S82 is compared with that of the designated image 1000. If the x- or y-coordinate value of the 
difference between the two positions is farther than a predetermined threshold value, a non-coincidence is determined, 
and the region of interest is excluded from the carKtidate regions. In this comparison, a value defined by Dp = IX - xl 
+ lY -yt ((x. y) is the baricentrtc coordinate position of the designated image 1000, and (X, Y) is that of the candidate 
region) is compared with the predetermined threshold value. 

For each of regions which are left as candidate regions in the processing operations in steps S81 to S83, a similarity 
(S) with the designated image 1000 is calculated in step S84 in accordance with the following equation: 

S = -{A"Dc + B^Ds + C»Dp) 
where A, B, and C are respectively weighting coefficients. 

In step S85, the plurality of similarity values (S) are sorted in the descending order. In step S86, K regions are 
^5 selected as candidate regions In the descending order from the maximum value, and their description information and 
similarity values are stored in the corresponding region buffer 26. 

With the above-mentbned processing, since the similarity values of K regions are obtained in association with the 
image (i), each similarity is represented by Srij in the following descriptbn. 

To summarize the above-mentioned processing, the number of candidates is decreased by excluding regions when 
^ the regions do not satisfy candidate conditions of corresporxiing regions In the color information discrimination process- 
ing in step S81 , the size Information discrimination processing in step S82. and the position information discrimination 
processing in step S83. The similarities between the remaining candidate regions and the designated Image 1000 are 
calculated in step S84, and are sorted In step S85. In step S86, K regions are selected as corresponding candidate 
regions in the descending order from the largest similarity value, and their description Information and similarity values 
^ are stored In the corresponding region buffer 26. 



•Designated Edge Presence/absence Discrimination Processing (Fig. 27) 



In this processing, whether or not an edge corresponding to the edge pattem of the designated image 1 000, which 
Is designated by the searcher, is present in an Image to be searched is checked with reference to the edge information 
stored in the designated Image memory 101, each corresponding candidate region information stored in the corre- 
sponding region buffer 26. and the description files 110, so as to calculate Information such as the presence ratb of 
edges, and the calculated Information Is output to the designated edge buffer 28. 

In the following description, a method of calculating designated edge information for each of K corresponding 
candidate regions (k: k = 1, 2,.,. K) will be explained. 

In step S91 , only line segments corresponding to edge information value = "1 * are extracted from the corresponding 
region description information stored In the corresponding regbn buffer 26 (the number of extracted line segments Is 
■m"). and are respectively defined as Ek1, Ek2,..., Ekm. On the other hand, in step S92, only line segments corre- 
sponding to edge informatbn value = "1 " are extracted from the designated image information stored in the designated 
image memory 101 (the number of extracted line segments is "n"), and are respectively defined as eki, ek2 ekn. 
Assume that all the line segments Eki , Ek2,..., Ekm, and eki , ek2 ekn have vertex coordinate data at the two ends. 

In step S93, ■+999999' Is set in dmin as an Initial value for calculating the minimum value (dmin) of distances 
between the two ends of the edge of the designated image and the line segment of a corresponding edge. In steps 
S94 to S98, a sum of the distances between the vertex coordinate positions at the two ends of a line segment eki of 
the designated image and each of all the edges (Eki Ekm) of the corresponding regions is cak;utated to obtain the 
minimum value (dmin) of the sum. 

In step S99, the minimum value (dmin) is compared with a predetermined threshold value (dth). If dmin < dth, the 
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flow advances to step S100 to determine that the line segment eki is present as an edge in the image to be searched, 
and *1 ' is set as the discrimination result. On the contrary, if dmin ^ dth, the flow advances to step SI 01 , and the line 
segment eki is not present as an edge in the image to be searched, and *0" is set as the discrimination result. 

The flow advances to step SI 02 to check if discriminatk>n of all the line segments eki of the designated image is 
5 completed. If NO in step S102, the flow advances to step SI 03 to increment the value i, and the flow returns to step 
S93 to continue the processing. On the other hand, if YES in step SI 02, the flow advances to step S104, and the 
obtained discriminatbn result informatbn is output as designated edge information of the candidate regions to the 
designated edge buffer 28. 

10 •Surrounding Region Determination Processing (Fig. 28) 

In this processing, the surrounding regions of each corresponding candidate regkxi are determined with reference 
to each corresponding candkJate regk>n infornnation stored in the corresponding regkxi buffer 26, and the features of 
each region of image data in the image files 109, whrch features are stored in the descriptbn files 110, and the de- 
ts scription informatk>n of the determined surrounding regk>n is output. 

In this case, the surrourwJing regk>n information of each of K corresponding candidate regions i = 2,..., K) in 
associatk)n with the image (i) is obtained. 

The corresponding region information is input from the corresponding regbn buffer 26 in step S11 1 , and the region 
description information of an image to be searched is input from the description files 110 in step S1 12. The flow then 
20 advances to step S113, and all neighboring regk>ns of the corresponding candidate region (j) are obtained. These 
regions are defined as the surrounding regions of the corresponding candidate region (j). In step S114, the color infor- 
mation of each of these surrounding regions is stored in the surrounding region buffer 30 as surrounding region infor- 
mation. 

25 •Total Similarity Calculation Processing (Fig. 29) 

In this processing, the similarities between the candidate regions and the designated innage 1000 are cak:ulated 
on the basis of the descriptKtn inforrr^tion of the corresponding candidate regk>ns stored in the corresponding regbn 
buffer 26, the designated edge information associated with the presence/absertce of the edges of the designated image 

30 1000, which information is stored in the designated edge buffer 28, and the surrounding regk>n informatk>n of the 
corresponding candklate regkxis stored in the surrouriding reg»n buffer 30, and the highest one of the calculated 
similarities is output to the sorting processing 128 as a total similarity between the designated image 1000 and image 
data in the image files 109. In other words, a carxikiate region of image data having the highest similarity obtained in 
this processing is determined as a region which best matches the designated image 1000 designated by the searcher, 

35 and its similarity is defined as one between the designated image 1 000 and the image to be searched. This processing 
is performed for all the image data in the image files 109, the similarities between the image data and the designated 
image 1000 are calculated, and the cak:ulated similarities are input to the sorting processing 128. 

In this processing, a total similarity TSij Q: j = 1, 2,..., K) of each corresponding candidate regbn (j: j = 1, 2,..., K) 
of the image (i) is cabulated. 

40 In step SI 21 , the initial value of the maximum value (Smax) of K total similarities (TSij) to be calculated is set. 

In step SI 22, the corresponding region information stored in the corresponding region buffer 26 is input. In step 
S123, the similarities (Srij) of corresponding regbns (j: j = 1, 2,..., K) are extracted from the input infomDatbn. 

In step SI 24, the surrounding region information stored in the surrounding regbn buffer 30 is input. In step S125, 
the similarity (Sbij) of the cobrs of the surrounding regions is calculated from the input infomnatbn using the folbwing 
45 equation: 

Sbij = -{IRbijI - rbl + IGbijI - gbl 
+ IBbijI - bbl + + IRbijn - rbl 
+ IGbijn - gbl + IBbijn - bbl}/n 

where (Rbiji , GbijI , BbijI ),..., (Rbijn, Gbijn, Bbijn) are the color informatbn values of the surrounding regions, and (rb, 
^ gb, and bb) is the surrounding cotor information value of the designated image 1 000. 

In step S126, the designated edge information stored in the designated edge buffer 28 is input. In step SI 27, the 
similarity (Seij) of the designated edge is calculated using the folbwing equatbn: 

Seij = (eO + el + + em)/m 
where (eO, el,.... em) is the designated edge information. 

In step SI 28, using the similarities cak;ulated In steps SI 22 to SI 27. a total similarity (TSij) is calculated based 
on the folbwing equation: 

TSij = [>Srij + E»Sbij + F»Seij 
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where D, E, and F are respectively weighting coefficients. 

When the above-mentk)ned processing is executed in combination with processing operations in steps SI 29 to 
S132, the total similarities (TSij) associated with the K corresponding candidate regions are calculated, and the max- 
imum value (Smax) of the K total similarities (TSij) is obtained. 
5 Finally, in step S133, a candidate region having the highest similarity, i.e., the maximum value, is determined as 

a region in the image (i), which region best n^atches the designated inrage 1000. 

The above-mentioned processing is executed for N images (i: i = 1 N) stored in the image file 109. The sorting 
processing 128 sorts the maximum similarities obtained in association with the respective images, and selects a pre- 
determined number of (K) image data in the descending order from the largest similarity value. Finally, the innage data 
10 access processing 129 accesses the image file 109 with reference to the similarity information of the predetermined 
numt>er of images so as to read out the corresponding K image data in the descending order from the largest similarity 
value, and displays the readout image data on the display 111. 

Therefore, according to the fourth embodiment, not only the region information of an image or figure pattern, but 
also the edge information of each of its regions and color information of surrounding regions are input as the designated 
15 image infornnation, and the number of image data to be searched is decreased using various kinds of description 
information of image data stored in the image files 109 on the basis of the input designated image information. Since 
only the remaining image data are accessed, the search processing of image data can be attained more accurately. 
In the search processing, since a large volume of data is not directly accessed, high-speed, efficient search processing 
can be realized. 

20 In the fourth embodiment, when the searcher generates a figure pattern of a designated image to be searched for, 

he or she draws the pattern on the screen of the display 111 by operating, e.g., the nrxsuse 8. However, the present 
invention is not limited to this. For example, the searcher may draw a sketch of a figure representing the designated 
image on an appropriate sheet, or may prepare for a sheet printed with a given image, and may load the image into 
the apparatus using an image input apparatus such as a scanner. The same applies to the above embodiments and 

25 those to be described below. 

When the searcher draws a figure, he or she draws the edges of the designated innage using, e.g., a pen on an 
appropriate sheet, and paints the designated image using, e.g., an appropriate pen. Then, the searcher paints the 
surrounding portion of the designated innage, and traces edges, which may be present in the image to be searched, 
of the designated image, using a predetermined pen. The sketch created as described above is read as the designated 

30 image using a scanner. The edge line of the read designated innage is polygonal-approximated using the linear ap- 
proximation processing method described in the fourth embodiment, and the average color in the designated image 
is determined as the cobr of the designated image. 

In the designated edge presence/absence discrimination processing described in the fourth embodiment, in the 
processing for discriminating whether or not an edge is present in an image to be searched, as shown in Fig. 27, the 

35 sums of the distances between the vertex point coordinate positions at the two ends of each line segment eki of the 

designated image and all the line segments Ekt, Ek2 E km of the regions of corresponding image data are calculated, 

and if the minimum value of the sunns is equal to or snnaller than the predetermined threshold value, it is determined 
that the line segment eki is an edge which is also present in the image to be searched. However, the present invention 
is not limited to this. For example, edge presence/absence discrimination processing shown in Fig. 30 may be per- 

40 formed. 

In step S1 41 , the coordinate positions and slopes of the central points of the line segments (edges) Eki , Ek2 Ekm 
of the innage to be searched and the line segments (edges) eki, ek2,..., ekn of the designated image are obtained. In 
step S142, the distance (dekiEkj) between the central point (ekic) of the designated edge (eki) and the central point of 
the edge (Ekj: j = 1,..., K) of the corresponding region of the innage to be searched is calculated. In steps SI 43 and 

45 SI 44, the edges (Ekj) of the corresponding regions, which have distances larger than a predetermined threshold value 
(dth), are excluded from the candidates of edges corresponding to the designated edge (eki). 

In steps S145toS147, the absolute value (d'ekiEkj) of the difference between eachof the slopes of all the remaining 
corresponding region edges (Ek€: 1 < € < m) which are not excluded, and that of the designated edge (eki) is calculated, 
and is compared with a predetermined threshold value (dih). If each absolute value is snnaller than the threshoki value, 

50 the flow advances to step SI 48; otherwise, the flow advances to step SI 49. In step S148, it is detennnined that the 
designated edge (eki) is present as an edge in the image to be searched, and "1 " is set as the discrimination result. 
In step SI 49, it is determined that the designated edge is not present as an edge in the image to be searched, and "0' 
is set as the discrimination result. 

As described above, according to the fourth embodiment, the number of image data to be searched is decreased 

55 using, as search designation information, not only an innage or figure pattem but also edges of the designated innage 
and the color of the surrounding regbns of the designated innage with reference to the feature data of innage data in 
the image file to be searched. For this reason, image data need not be directly accessed to decrease the number of 
innage data, and a desired image can be accurately found at higher speed. 
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[Fifth Embodiment] 

The fifth embodiment of the present invention will be described below. In this embodiment, a designated innage is 
input by designating an edge line and its surrounding color 
s Fig. 31 is a block diagram showing the arrangement of an image database system according to the fifth embodiment 

of the present invention. 

Referring to Fig. 31, reference numeral 131 denotes an image database system main body; 132, an edge line 
input unit for instructing search processing; 133, an edge surrounding region generation unit for generating an edge 
surrounding region shown in, e.g., Fig. 35, on the basis of an edge line input from the edge line input unit 132; and 

10 134, a color input unit for inputting the color of each surrounding region generated by the edge surrounding region 
generatbn unit 133 by a searcher. With these units, a desigr^ted image as a search key is input. Reference numeral 
111 denotes a display for displaying the search result. Note that the edge line input unit 132 ar\6 the cobr input unit 
1 34 correspond to the figure input window 201 and the cotor palette 205 shown in Fig. 3. 

In the image database system 131 , the innage file 109 stores image data to be searched. An edge image memory 

IS 135 stores edge image data generated by performing edge extraction processing for image data stored in the image 
file 1 09 in advance. Reference numeral 1 36 denotes a candidate edge surrounding region generation unit for generating 
a region (to be referred to as a candidate edge surrounding region hereinafter) corresponding to the edge surrounding 
region generated by the edge surrounding region generation unit 133 on the basis of an original image stored in the 
image file 109 and an edge image stored in the edge image memory 135. 

20 A distance calculation unit 136 calculates the distance between the edge surrounding region generated by the 

edge surrounding region generatbn unit 1 33 and the candidate edge surrounding region. A display image selectbn 
unit 1 37 selects a (similar) stored image having a short distance to the designated image, which distance is calculated 
by the distance calculation unit 136 with respect to the respective stored images in the image file 109, and displays 
the selected image on the display 111. 

25 In the system according to the fifth embodiment with the arrangement shown in Fig. 31, upon storage of innage 

data in the image file 1 09, the image data is input to the image file 109, and at the same time, an edge innage obtained 
by performing known edge detection processing for the image data is stored in the edge image memory 1 35. More 
specifically, this image database system 131 stores original image data and edge image data corresponding to each 
original image data. Note that edge image data is stored after it is binarized by a known method. 

30 In the fifth embodiment, the image search method from the innage database uses, as a designated image, the 

position of an edge portion of an image and the surrouruiing color of the innage in a searcher's mennory which are 
designated by the searcher. 

The retrieval processing for retrieving images stored in the image database in the fifth embodiment will be described 
below with reference to the flow chart shown in Fig. 32. 
35 In step S151 , the searcher draws a polygonal line on the figure input window 201 using a graphic editor described 

above with reference to Fig. 3 so as to input a designated image. For example, when the searcher wants to search tor 
an innage shown in Fig. 33. he or she inputs line segments corresponding to a clear edge portion in the image data as 
a polygonal line on the basis of his or her memory, as indicated by reference symbols A, B, C, and D in Fig. 34. If the 
edge of an original image is approximate to a straight tine, the edge nnay be designated by a straight line (the polygonal 
40 line will be referred to as an edge line hereinafter). 

In step SI 52. image dilation processing of x dots (x is a predetermined value) is performed for the edge line 
designated in step SI 51 to obtain a region E shown in Fig. 35. The region E is divided by straight lines which pass the 
two end points (A and D in Fig. 34) of the input edge line arid are perpendicular to segments (AB and CD in Fig. 34) 
including the two end points, and a portion, including the designated edge line, of the divkied partial regions is deter- 
45 mined as an edge surrounding region. 

As a result, the edge surrounding regbn is divided into two regions X and Y by the edge line, as shown in Fig. 36. 
Upon observation of the edge line from its input start point (A in this case) to its input end point (D in this case), the 
left one of the two regions is defined as the regkxi X, and the right region is defined as the region Y 

In step SI 53, the edge surrounding regions obtained in step SI 52 are displayed on the display 111 . The searcher 
50 inputs the cobrs of portbns, corresponding to these regk>ns, of the original image to one or both the two regkxis X 
and Y using the color input unit 1 34 according to his or her memory. 

In steps SI 54 to S160 (processing with symbol A in Fig. 32), the processing to be described below is performed 
for each image stored in the image database. 

More specifically, in step SI 54, only portions corresponding to the edge surrounding regions obtained in step S1 52 
55 are extracted from the edge images stored in the edge image mennory 135. For example, assume that an edge sur- 
rounding region having a shape shown in Fig. 37A and an edge innage shown in Fig. 37B are present. The region 
shown in Fig. 37 A and the image shown in Fig. 37B are superposed each other, and portions, indbated by solid lines 
in Fig. 37D, of the obtained edge (Fig. 37C) are extracted. Note that the extracted portions will be referred to as 
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corresponding edge candidates hereinafter. 

In step SI 55, the extracted corresponding edge candidates are subjected to binary image dilation processing by 
y dots (y Is a predetermined value, as shown in Fig. 37E, thus combining ones, which are close to each other, of the 
corresponding edge candidates. In step S156, the regions subjected to the inflation processing in step S155 are then 
s subjected to known thin-line conversion processing to be converted into continuous polygonal lines having a thickness 
"1" and branches, as shown in Fig. 37F 

In step SI 57, the polygonal lines obtained in step SI 56 are traced by a method of generating chain codes, thus 
extracting the longest chain, as shown in Fig. 37G. The extracted chain is subjected to polygonal approximation by 
the divisional linear approximation method, so that the number of segments of the extracted chain beconries equal to 
10 the number of (three in Fig. 34) of segments included in the input poly^al line. 

Fig. 37H shows the result of the above-mentioned polygonal approximation. The obtained polygonal line (A' - D') 
senses as a candidate of an edge corresponding to the input edge line shown in Fig. 34. Note that this polygonal line 
will be referred to as a candidate edge line hereinafter. 

As for the two erKl points of the candidate edge line, a point A' close to the input start point A of the input edge 
15 line is assumed to correspond to the input start point. In this case, the other end point of the candidate edge line 
corresponds to the end point (D') of the input edge line. 

In step S158, two edge surrounding regions are generated from the candklate edge line obtained in step S157 in 
the same manner as in the processing in step S1 52, as shown in Fig. 371. These regions will be referred to as candidate 
edge surrounding regions hereinafter. Upon observation of the candidate edge line from its end point (A') corresponding 
20 to the start point (A) of the input edge line to the other end point (D*), the left one of the two candidate edge surrounding 
regions is defined as a region X', and the right candidate edge surrounding regk^n is defined as a region Y'. The regions 
X' and Y* respectively correspond to the regions X and Y in Fig. 36. 

In step S159, the representative cok>rs of regions, corresponding to the candkiate edge surrounding regions ob- 
tained in the processing until step S158, of the stored original image are obtained. Note that various methods of ob- 
2S taining the representative color of a given regkjn are known. In this case, the averages of R, G, and B values of all the 
pixels in the regk>n may be simply cak^ulated. 

In step S160, the distances between the colors input by the searcher to the edge surrounding regions in step S1 53 
and the representative cotors of the con-esponding candkJate edge surrounding regbns are cateulated. Some color 
distance calculatbn methods are known. In this case, the distance between the two colors in the RGB space may be 
30 simply cateulated. When a color is input to only one of the edge surrounding regtens in step S153, a similarity is 
cateulated using only the regkxi and the corresponding candkiate edge surrounding regkxi. On the other hand, when 
cokxs are input to both the regions X and Y in step S153, similarities are cateulated between the edge surrounding 
regions and the corresponding candidate edge surrounding regtens, and the calculated values are averaged. In this 
case, this average value is used as the distance between the input value and the stored inrage. 
35 The above-mentioned processing in steps SI 54 to SI 60 is repeated for all the stored images in the image file 1 09, 

thus obtaining the distances between the input value and the stored images. In step SI 61, stored images having 
smallest distances (highest similarities) are selected, and are displayed on the display 111 to be presented to the 
searcher as the search results. 

As described above, according to the fifth embodiments, even when a target image does not have any equal-coter 
40 region of certain size, as teng as the target image includes a clear edge, the searcher designates the edge by a straight 
or polygonal line, and also designates the coters of one or both surrourKiing regons obtained by inflating line segments 
of the designated edge line to input a designated image. tJsing the desigrtated image, an image search operation from 
the image database (image file) is performed, thus facilitating the input operation of the designated image and allowing 
a quick search. 

46 Note that the present invention is not limited to the above embodiment. For example, matching between the can- 

didate edge line obtained in the processing in step SI 57 in Fig. 32 and the input edge line may be performed, and if 
it is difficult to attain matching, i.e., if the two edge lines are not similar to each other at all, the corresponding stored 
image may be excluded from data to be searched, thus realizing higher-speed search processing. 

As the matching method, for example, as shown in Fig. 38, when the start point (A) of the input edge line is 
so superposed on the corresponding end point (A') of the candidate edge line, if the ratte of the distance, d, between the 
remaining end points (D and D') of the two edge lines to the length of the input edge line is larger than a predetermined 
value, a source image of the candidate edge line may be excluded from data to be searched. 

In the fifth embodiment, an edge which is input by the searcher based on his or her memory associated with an 
original image need not be a polygonal line. For example, a cun^e shown in Fig. 39 may be input with respect to the 
55 image (an image to be searched) shown in Fig. 33. In this case, the input curve may be subjected to the divistenal 
linear approximation method, and the same processing as described above may be performed using a polygonal line 
obtained by approxinrratwn of the cun/e as the input edge line, as shown in Fig. 40. 

In this case, unlike in the divisbnal linear approximation method in the fifth embodiment, the processing is repeated 
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until the distance between the input curve and the polygonal line becomes smaller than a predetemriined value. 

As described above, according to the fifth embodiment, an Image search operation can be quickly and reliabty 
performed on the basis of a comparison of the position of the input edge line and the colors designated on its surrounding 
regions with stored image data. 

[Sixth Embodiment] 

Fig. 41 is a block diagram showing the arrangement of an image retrieval apparatus according to the sixth em- 
bodiment. 

Referring to Fig. 41, reference numeral 211 denotes a feature amount calculaton unit for calculating feature 
amounts corresponding to a plurality of block sizes shown In Figs. 42A to 42C from respective image data (stored in 
the image file 109) to be searched; and 212, an index generation unit for generating indices for search based on the 
feature amounts cateulated by the feature amount calculation unit 211. Reference numeral 213 denotes a resolution 
Index table formed in units of resolutions; and 214, a table search unit for determining the size and feature amount of 
a block to be searched on the basis of the Input designated inrrage 100 (see Fig. 5 or 24), arxJ searching for a similar 
image and corresponding regions on the basis of the index table 213. 

In the above arrangement, the feature amount cafculation unit 211 cateulates and outputs a plurality of feature 
amounts corresponding to a plurality of btock sizes for each innage data stored in the image file 109. The index gen- 
eration unit 212 generates Indices based on the feature amounts output from the feature amount cak:ulation unit 211, 
and search informatbn (to be described later) is stored in the resolution index table 213 in correspondence with the 
indices and resolutbns (bbck sizes). 

Upon search of inr\age data In the image file 109, the table search unit 214 calculates the block size and feature 
amount to be searched for on the basis of the designated Image 100, searches the Index table 213 using them, and 
outputs candidate images and their corresponding regions as retrieval results. 

Figs. 42A to 42C show examples of a plurality of block sizes used in the feature anrKDunt cateulatlon unit 211 . 

Fig. 42A shows 16x16 pixel data per block, Fig. 42B shows 32 x 32 pixel data per block, and Fig. 42C shows 64 
x 64 pixel data per block. In this case, a block size = 16 corresponds to the highest resolution. 

The feature amount cateulatbn unit 211 calculates the color histogram of pixel data In units of blocks for each 
resolution (block size), and outputs a cotor feature vector (CI, C2, C3,..., Cn). 

Fig. 43 is a view for explaining the process for cateulating the cok>r feature vector from the histogram. Cotor data 
for which a histogram is calculated can be cateulated using varwus cotorimetric systems. In this case, an HSV colori- 
metrlc system is used as an example, and the histogram of H data representing a hue is used. Each element CI of the 
color feature vector is obtained by counting the number of pixels within the range of [di - 1, di + 1] in the histogram. 
Since elements Ci and CI+1 have an overlapping portion, even when the feature value obtained from the designated 
Image 100 is in the neighborhood of a quantization step di, stable discrimination is assured. 

Fig. 44 Is a flow chart showing the operation of the index generation unit 21 2. 

This flow includes simple triple loops. In step S171, a resolutton (block size) R is set. In step S172, the block 
number j is initialized to "1". In step SI 73, a parameter i IrxJIcating each element Ci of the cotor feature vector is 
initialized to M Subsequently, in step SI 74, the element CI of the cotor feature vector is compared with a threshold 
value T(R) determined based on the resolution R. As a result, if the element CI is larger than the threshold value T(R), 
the flow advances to step S175, and the block number (region id) and the number (image Id) of an image including 
the block are added to a column, determined by the resolution R and the cotor feature vector element CI, of the Index 
table 213. In steps SI 76 to S181, the respective parameters are updated, and when indices of elements Ci are gen- 
erated for btocks of all the resolutions, the processing ends. 

Fig. 45 is a view showing the arrangement of the index table 213. When the operation shown in Fig. 44 Is repeated 
for all the resolutions (block sizes) R and all the btocks j, index Informatton to In^ages and regtons having a predeter- 
mined frequency or higher of a specific cotor Is stored in the index table 21 3 in units of blocks. 

Fig. 46 is a flow chart showing the operation of the table search unit 214. 

In step S201 , a hue value H and a corresponding color range Ch are calculated on the basis of color data of the 
designated image 100 Input as a pattern to be searched for, as shown In, e.g., Fig. 5, In step S202, a block size to be 
searched for is determined on the basis of the shape of the designated image 100. Note that determination of the btock 
size (s) will be described later with referertce to Fig. 47. 

In step 8203, the index table 21 3 is searched using the block size determined in step S202, thus obtaining the 
candidate image number (Image Id) and search btock number (regton Id). The hue value H Is obtained as follows. That 
Is, the hue value H is compared with each threshoto value di In accordance with the relationship shown in Fig. 43, and 
when the hue value H satisfies: 

dk < H < dk + 1 
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elements Ck and Ck+1 are set as an appropriate color range. 

Fig. 47 is a flow chart showing the processing tor determining the aboveH7ientk)ned block size (s). 

In step S211 , a maximum height h, a maximum width w, and an area (the number of pixels in the designated image) 
region are calculated for the designated image 100. In step S212, a ctosest block size having a height and width equal 
s to or larger than h and w is set to be an initial value of s. In step S213, it is checked if the area s x s of the block is 
larger than the product of the area region (area) of the designated inrtage 100 and a predetermined constant T If YES 
in step S21 3, the flow advances to step S214, and s is multiplied with 1/2. Then, the flow returns to step S21 3 to repeat 
comparison. On the other hand, if NO in step S213, the value s is determined as a block size for the next search. Note 
that the constant T is determined in association with the threshokl value T(R) upon generation of an index, and assumes, 
10 e.g., a value "4.0". 

When the image search apparatus according to the sixth embodiment is applied to an image database system, 
an electronic filing system, or a multimedia database system, a system with a high-speed image search function can 
be provided. 

In the sixth embodiment, the feature amount cabulation unit 211 cak:ulates the cobr feature vector in correspond- 
15 ence with the bkx:k sizes shown in Figs. 42A to 42C. However, the present invention Is not limited to these bkjck 
formats. For example, the block formats shown in Figs. 48 A and 48B may be used. 

In Figs. 48 A and 48B, the moving anrK>unt of a block with respect to a bkxk size 2A is set to be a value different 
from the block size, e.g., A. More specifically, blocks 1a(1, 1), 1a(1, 2), 1a(2, 1) and la(2, 2) shown in Fig. 48Aof the 
block size A are set to be a block 1(1, 1) in Fig. 48B of the block size 2A, and blocks 1a(1, 2), 1a(1, 3), 1a(2, 2), and 
20 ia(2, 3) partially overlapping the bkxk 1(1. 1)are set to be a block 1(1, 2). Bkx:ks are formed by similarly permitting 
overlapping of blocks in both the vertical and horizontal directions. 

With these formats, although the number of entries of the index table 21 3 increases, the influence caused when 
the designated image 100 overlaps bkx:ks can be eliminated, and the efficiency upon use of indices can be improved. 

The retrieval operation of the sixth embodiment may be used as a primary screening operatbn, and selected 
^5 candidates may be subjected to comparison in more detail using methods in first and fifth embodiment or a conven- 
tionally proposed technque, thus obtaining retrieval results with higher accuracy. 

As described above, according to the sixth embodiment, even when the designated image is a portion of a target 
image, the apparatus can operate stably, and the number of images to be compared can be sufficiently reduced, thus 
realizing high-speed image search processing. 

30 

[Seventh Embodiment] 



Fig. 49 is a bkx:k diagram showing the funct»nal arrangement of an image retrieval apparatus according to the 
seventh embodiment of the present invention. 

35 Referring to Fig. 49, reference numeral 221 denotes an image registration processing unit for performing pre- 

processing (to be described later) before an image is registered in an image database system 222. Reference numeral 
222 denotes an image database system; and 223. an image (designated image) input unit for instructing the image 
retrieval apparatus of the seventh embodiment to start search. The unit 223 is used for inputting the designated image, 
as shown in Fig. 3 above. Reference numeral 111 denotes a display for displaying the retrieval results. 

40 The arrangement of the image registratbn processing unit 221 will be described betow. Reference numeral 231 

denotes a registered image input sectk»i for inputting an image to be registered in the image file 109 of the database 
system 222. Reference numeral 232 denotes a region division processing section for dividing the image input from the 
registered image input sectbn 231 into regk>ns with simitar cobrs. Reference numeral 233 denotes a region feature 
amount extraction sectbn for extracting various feature anrK>unts (cobrs, shapes, and the like) in units of regions divided 

45 by the region division processing section 232. Reference numeral 234 denotes a color index generation section for 
generating a cobr index of the registered image on the basis of cobr data in units of regions extracted by the region 
feature amount extraction section 233. 

The arrangement of the image database system 222 will be described bebw. 

The image file 1 09 stores a registered image input from the registered image input sectbn 231 . Reference numeral 
50 242 denotes a region feature amount storage section for storing feature amounts, extracted by the region feature 
amount extraction section 233, of regbns obtained by dividing the registered image in units of regions of each registered 
image. Reference numeral 243 denotes a cobr index storage section for storing the color index of each registered 
image generated by the cobr index generation section 234. Reference numeral 244 denotes a screening processing 
section for comparing the designated image, which is input by a searcher from the designated image input unit 223 to 
S5 the image retrieval apparatus of the seventh embodiment, with the cobr index, and determining whether or not the 
next region similarity calculatbn sectbn 245 performs a cabulation of a similarity for the region of interest. The region 
similarity calculation section 245 performs matching between the corresponding regbns of the designated image and 
the stored images, which are not excluded from processing targets by the screening processing of the screening 
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processing section 244, so as to calculate their sinriilarities. Reference numeral 246 denotes an intermediate result 
buffer for temporarily storing the calculation results of similarities in units of images in the region similarity calculation 
section 245. Reference numeral 247 denotes a candidate order determination section for selecting images with higher 
similarities with reference to the similarities in units of images stored in the intermediate result buffer 246, and displaying 
s the selected images on the display 111. 

As an image retrieval method from the image database, the seventh embodiment exemplifies a case wherein at 
least one closed region which is drawn by a searcher (operator) by designating its position, shape, and color is used 
as an image serving as a search key, and an image stored in the image database is searched using the position, shape, 
and color of the closed region. 

10 Fig. 51 is a flow chart showing the processing upon registration of an image in the image database system 222 in 

the image retrieval apparatus of the seventh embodiment 

In step S221 , an image to be registered in the image database system 222 is input from the registered image input 
section 231 , and is stored in the image file 109. The flow then advances to step S222, and the input image is divided 
into near-equal-color regions by the segmentation processing section 232. 

IS The flow advances to step S223, and various feature amounts are extracted in units of regions obtained by dividing 

the input image into near-equal-color regions in step S222. The extraction results are stored as region feature amounts 
in the region feature arDOunt storage section 242 in units of input images. The region feature amounts include the 
representative color (H, S, and V values), area, and shape (the coordinate positions of the vertices of a polygon obtained 
by polygonal approximation of a region) of each region. The flow then advarK:es to step S224, and a color index is 

20 generated on the basis of the region division results of the Input innage. 

A method of generating a color index in the color index generation section 234 of the seventh embodiment will be 
described below. 

The representative colors of the respective regions extracted in step S223 are mapped in the HSV color space. 
In this case, the areas (each value indicates the ratio to the area of the entire image, and the 'area' means this ratio 
25 in the folbwing description) of the corresporKiing areas are stored at the mapped coordinate positions. The mapped 
color space is determined as a color index, and is stored in the color index storage section 243 in units of stored images. 
In this manner, the processing upon image registration is completed. 

The processing upon search of an image registered in the image database system 222 will be described betow. 
Fig. 52 is a flow chart showing the image search processing in the image retrieval apparatus of the seventh em- 
30 bodiment. 

In step S231, the searcher inputs an tnnage or figure (designated innage) as a search key from the designated 
image input unit 223 by handwriting by means of, e.g., a drawing editor or using a scanner. For example, when the 
searcher wants to search for an image shown in Fig. 50A. he or she inputs an equal-cotor regbn in the image as one 
closed figure 200, as shown in, e.g.. Fig. 50C, by designating its shape and cotor. This figure will be referred to as a 

35 designated image hereinafter. 

The flow advances to step S232, and the area, Sc, of the designated closed figure 200 is cateulated on the basis 
of the designated image shown in Fig. 50C. This area is calculated as the ratio of the area of the closed figure 200 to 
the entire designated image, as described above. The flow advances to step S233, and a color index, which has already 
been generated for a certain one of images stored in the image files 109, is read out from the cobr index storage 

40 sectk>n 243. 

The flow advances to step S234, and the cotor of the designated image is converted into H, S, and V values, and 
the converted values are mapped on the same cokx space as the color irKiex loaded in step S233, as shown in Fig. 53. 

Fig. 53 is a view for explaining comparison between the cok>rs of the respective regions in a stored image and the 
color of the designated image. Referring to Fig. 53, Pi is the coordinate position, in the color space, of the cotor of a 
45 region i in the stored image, and C is the coordinate positkxi, in the color space, of the cotor of the designated image. 
Si is the area of the region i. 

The flow then advances to step S235. In step S235. a spherical surface 600 of a radius r (predetermined value) 
is drawn on this cotor space to have C as the center, and the coordinate posit ton Pi included in this sphere is searched 
for. Then, the areas Si of all the coordinate positions Pi included in the spherical surface 600 are calculated, and a 
50 total Sp of the areas Si is calculated. 

The flow advances to step S236 to calculate Sp (total of Si)/Sc (the area of the closed figure 200), and if the 
calculated value is smaller than a predetermined value x (< 1 ), i.e., if *the area of the designated image' is considerably 
larger than the total of the areas of the regions having cotors similar to that of the designated image', it is determined 
that 'an equal-color regton of the designated image that the searcher intended is not present in the corresponding 
55 stored image", i.e., the stored image is quite different from the image that the searcher intended". Thus, the stored 
image is excluded from data to be searched in the subsequent search processing. The flow then advances to step 
S237 to load the color index of the next stored image, and the flow returns to step S234. 

If the cateulated value is not smaller than the predetermined value x, the flow advances to step S238. In step S238, 
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the region feature amounts of stored images, which are nd excluded from processing targets in step S236 are read 
out from the regbn feature amount storage section 242, and the region similarity calculation section 245 performs the 
matching discrimination processing between the regions and the designated inrtage. The similarity values between the 
designated image and the stored images obtained as a result of the nriatching discrimination processing are stored in 

s the intermediate result buffer 246 in units of stored images. 

The flow advances to step S239 to check if the above-mentioned search processing is completed for all the stored 
images stored in the image file 109. If NO in step S239, the flow retums to step S237 to start the processing for the 
next stored image. In this manner, when the processing for all the stored images in the image file 109 Is completed, 
the flow advances to step S240, the candidate order determination section 247 sorts the similarities (stored in the 

10 intermediate result buffer 246) between the stored images and the designated image, and displays images having 
higher similarities of those subjected to the region matching processing on the display 111, thus ending the retrieval 
processing. 

With the above-mentioned processing, the number of stored inr^ges to be subjected to the region matching 
processing which requires a long processing time can be efficiently decreased, and as a result, the search time can 
IS be shortened. 

[Eighth Embodiment] 

In the seventh embodiment described above, a spherical surface is drawn on the color space of the color index to 
20 have the color of the designated image as the center, and a total of the areas of regions in the stored image having 
colors included in the spherical surface is calculated. However, since this processing requires a relatively long search 
time, the search time can also be shortened by simplifying the processing as follows. 

Upon generation of a color index for each stored image, the HSV color space is divided into appropriate blocks, 
as shown in Fig. 54A, a total of the areas of regions having colors included in each block is calculated in units of blocks, 
25 and the calculated value (Tj for a block j) is stored as a cotor index in the cotor index storage section 243 in units of blocks. 

Upon retrieval of an image, the cotor of the designated inrage is mapped on the color space as in the seventh 
embodiment described above, and a total of the values Tj of all the blocks included within a range of a predetermined 
value r2 from the mapped coordinate position is defined as Tp. If a value Tp/Sc is smaller than a predetemnined value 
x2 (< 1), it is determined that the stored image is quite different from the image that the searcher intended, and the 
30 stored image is excluded from data to be searched in the subsequent processing as in the above embodiment. 

[Ninth Embodiment] 

In the seventh and eighth embodinnents, the designated image fs input as a figure having one closed region. 
35 However, the present invention is not limited to this. For example, the designated image nnay be input as a plurality of 
closed regions. 

In such a case, the same processing as in the above-ment toned processing up to step S237 in the flow chart of 
Fig. 52 in the seventh embodiment is performed for each closed region, and the processing in step S238 and the 
subsequent steps can be performed for only stored images which are not excluded from data to be searched in asso- 

40 ciatton with all the closed regtons. 

As described above, according to the seventh embodiment, using data of the area of a ctosed region (designated 
image) designated as an image serving as a search key, the screening operatton for decreasing the number of innages 
to be subjected to the subsequent region matching processing can be performed more effectively, and as a result, the 
search time can be greatly shortened. 

45 The present invention is not limited to the above embodiments and various changes and modifications can be 

made within the spirit and scope of the present invent ton. Therefore, to apprise the public of the scope of the present 
invention, the following claims are made. 



50 Claims 

1. An image retrieval apparatus for retrieving image data including an image similar to a designated image to be 
retrieved using, as reference information, image description information which describes information for indivtoual 
regions in the image, comprising: 

55 

single region processing means for detecting a single regton included in the designated image from the indi- 
vidual regions in accordance with the individual regions in the image and the designated image to be searched 
ton 
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compoLind region processing means for obtaining a compound region as a combination of the single regions 
included in the designated image in accordance with combinations of the single regions and the designated 
image; 

similarity detection means for calculating a similarity between the compound regbn and the designated image; 
s and 

search means for searching image data in accordance with the similarity calculated by said similarity detection 
means. 

2. The apparatus according to claim 1 . wherein said single region processing means discriminates the corresponding 
10 single region in accordance with a color information difference between each of the individual regions in the image 

and the designated image, and shape information of the designated image. 

3. The apparatus according to claim 1 , wherein said single regk^n processing means discriminates the single region 
depending on whether or not each of the individual regions in the image has a predetermined positional relationship 

15 with the designated image. 

4. The apparatus according to claim 1 , wherein said compound region processing means discriminates the compound 
region on the basis of a crossing region of geometric parameter regions indicating a range in which each of the 
irKJividual regions in the image has a predetermined positional relationship with the designated image. 

20 

5. The apparatus according to claim 1 , wherein said compound region processing means discriminates the compound 
region depending on whether or a crossing region of geometric parameter regions indicating a range in which each 
of the individual regions in the image has a predetermined positbnal relationship with the designated image falls 
with in a variable range of the designated image. 

25 

6. The apparatus according to claim 1 , wherein said single region processing means comprises means for comparing 
a color of each of the individual regions in the image with a color of individual regions in the designated image, 
and detects the single region by discriminating whether or not each of the individual regions in the image has a 
predetermined positional relationship with the designated image. 

30 

7. An image retrieval apparatus for retrieving tnnage data stored in a file on the basis of externally input information, 
comprising: 

input means for inputting figure infonmation, and edge ffiformation and surrounding color information of the 
35 figure information, as retrieval information; 

storage means for storing feature data of image data stored in the file; 

extraction means for extracting image data serving as a search candidate on the basis of the figure information 
input from said input means and the feature data; 

first generation means for generating first help information for decreasing the number of search candidates 
40 on the basis of the edge infonmation of the figure information, the feature data, and an extraction result of said 

extraction means; 

second generation means for generating second help information for decreasing the number of search can- 
didates on the basis of the surrounding color information of the figure information, the feature data, and the 
extraction result of said extraction means; 
45 determination means for determining a target image on the basis of the extraction result of said extraction 

means, the first help information, and the second help information; 

access means for reading out the target image by accessing the file on the basis of a determination result of 
said determination means; and 

display means for displaying target irr^ge data read out by said access means. 

so 

8. The apparatus according to claim 7, wherein said input means comprises at least a keyboard, a pointing device, 
and a display. 

9. The apparatus according to claim 7, wherein the image data stored in the file is managed while being divided into 
55 a plurality of regions, and said extraction means extracts an image sen/ing as a search candidate by discriminating 

which one of the plurality of regions serves as the search candidate. 

10. The apparatus according to claim 7. wherein said determination means quantifies and evaluates the extraction 
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result of said extraction means, the first help infornnation, and the second help information so as to systematically 
determine the image data serving as the search candidate, and determines one or a plurality of image data as a 
target image or inrtages on the basis of the quantified values. 

s 11. The apparatus according to claim 10, further comprising sort means for sorting the quantified values in a descending 
order when said determination means determines the plurality of innage data as search candidates. 

12. The apparatus according to claim 7, wherein said input means can input color information of the figure information. 

10 13. An image retrieval apparatus for searching a plurality of image data stored in a database for a desired image, 
comprising: 

means for inputting an edge tine of a designated image which is designated to be retrieved from images to be 
searched; 

15 means for designating a color for a surrounding region of the edge line; 

means for calculating similarities between the designated image and the plurality of image data on the basis 
of the edge line and the cobr of the surrounding region; and 

means for displaying an image corresporKiing to image data having a maximum similarity as a target image. 

20 14. The apparatus according to claim 13, further comprising calculation means for calculating distances between the 
designated Image and the plurality of innage data, and wherein the distance corresponds to the similarity between 
the designated image and the image data. 

15. The apparatus according to claim 14, wherein said calculation means comprises: 

25 

means for storing edge images corresponding to the plurality of innage data; and 

means for extracting an edge of a portion corresponding to the edge line from the edge innages, and 

a distance between a color of a surrounding region of the extracted edge and the designated color corresporrds 

to the distance. 

30 

16. The apparatus according to claim 13, wherein the edge tine is a polygonal line. 

17. The apparatus according to claim 13, wherein the edge line is a curve. 

35 18. An innage retrieval apparatus for retrieving an image similar to an input pattern, comprising: 

calculation means for calculating a feature amount of image data in correspondence with a plurality of reso- 
lutions; 

generation means for generating search information on the basis of the feature amount calculated by said 
40 calculation means; 

storage means for storing the search information generated by said generation means in units of resolutions; 
and 

retrieval means for retrieving an image similar to an input pattern on the basis of the search information stored 
in said storage means. 

45 

19. The apparatus according to claim 18, further comprising resolution determination means for determining a reso- 
lution used for search on the basis of vertical and horizontal dimensions and an area of each of regions in the input 
pattern. 

50 20. The apparatus according to claim 1 8, wherein said calculation means calculates the feature annount by calculating 
a cobr histogram of pixel data in units of resolutions. 

21. The apparatus according to claim 20. wherein the color histogram is a histogram of a hue in an HSV colorimetric 
system. 

55 

22. The apparatus according to claim 1 8, wherein the feature amount is calculated on the basts of the number of pixels 
within a predetermined hue overlapping range. 
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23. The apparatus according to claim 18, wherein the search Information includes numbers of an Image and a region 
having not less than a predetermined frequency of a specific color at each of the resolutions. 

24. An image retrieval apparatus for searching a plurality of stored images for a desired image, comprising: 

5 

input means for inputting an image to be retrieved including a closed region; 

comparison means for comparing colors of regions obtained by dividing each of the stored images with a color 
of the closed region of the innage to be retrieved, and comparing a total of areas of the regions having approx- 
imate colors of the stored image with an area of the cfosed region; 
10 determination means for determining, as an image to be retrieved, a stored image in which a total of the areas 

of the regions is not considerably smaller than the area of the closed region as a result of comparison of said 
comparison means; 

similarity discrimination means for calculating a similarity by discriminating matching between the stored image 
determined as the image to be retrieved by said determination means, and the image to be retrieved; and 
15 display means for displaying the image which is discriminated by said similarity discrimination means to have 

a similarity not less than a predetermined value as a retrieval result. 

25. The apparatus according to claim 24, wherein the plurality of stored images are stored in such a manner that each 
of the plurality of stored images is divided into a plurality of regions, and sets of colors and areas of the plurality 

20 of regions are mapped on a color space. 

26. The apparatus according to claim 24. wherein the image input from said Input means Includes the closed region 
having a color, a position, and a shape. 

26 27. The apparatus according to claim 24. wherein said display means displays target Images by sorting the images 
in a descending order from the highest similarity. 

28. An image retrieval method for retrieving image data including an Image similar to a designated image to be retrieved 
using, as reference lnformatk)n. Image description Information which describes Information for individual regions 

30 In the image, comprising the steps of: 

detecting a single region included in the designated image from the Individual regions in accordance with the 
Individual regions in the image and the designated image to be searched for; 

obtaining a compound region as a combination of the single regions Included in the designated image in 
35 accordance with combinations of the single regions and the designated Image; 

calculating a similarity between the compound region and the designated Image; and 
searching image data in accordance with the calculated similarity 

29. The method according to claim 28, wherein the single region is discriminated in accordance with a color Information 
40 difference between each of the individual regions in the image and the designated image, and shape inf omration 

of the designated image. 

30. The apparatus according to claim 24, wherein the single region is discriminated depending on whether or not each 
of the individual regions in the Image has a predetermined positional relationship with the designated image. 

45 

31. The apparatus according to claim 24, wherein the compound region is discriminated on the basis of a crossing 
region of geometric parameter regions indicating a range In which each of the Individual regions in the Image has 
a predetermined positional relationship with the designated Image. 

50 32. An Image processing method capable of retrieving image data stored in a file on the basis of externally input 
information and feature data of the image data stored in the file, comprising the steps of: 

Inputting a pattem representing one of an image and a figure, and edge infonDation and surrounding color 
information of the pattem, as search Infonmation; 
55 extracting image data sen/Ing as a search candidate on the basis of the pattern and the feature data; 

generating first help Information for decreasing the number of search candidates on the basis of the edge 

information of the pattem, the feature data, and an extraction result In the extracting step; 

generating second help information for decreasing the number of search candidates on the basis of the sur- 
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rounding color information of the pattern, the feature data, and the extraction results in the extracting step; 
systernatically determining image data as the search candidate on the basis of the extraction result in the 
extracting step, the first help infornnation, and the second help information; 

reading out the image data by accessing the file on the basis of a detemriination result in the determining step; 
5 and 

the display step of displaying the image data read out in the reading step. 

33. An image retrieval method for searching a plurality of Image data stored in a database for a desired image, com- 
prising the steps of: 

10 

inputting an edge line of a sample image for an innage to be retrieved; 
designating a color for a surrounding region of the edge line; 

calculating similarities between the sample image and the plurality of innage data on the basis of the edge line 
and the color of the surrounding region; and 

displaying an image corresponding to image data having a maximum similarity as a target image. 

34. The method according to claim 33, further comprising the calculation step of calculating distance values between 
the sample image and the plurality of image data, and wherein the distance value corresponds to the similarity. 



20 35. The method according to claim 34, wherein the calculation step comprises the steps of; 

storing edge images corresponding to the plurality of inoage data; and 
extracting an edge of a portion corresponding to the edge line f rcxn the edge images, and 
a distance between a cobr of a surrounding region of the extracted edge and the designated color correspond- 
2^ ing to the distance value. 

36. The method according to claim 33, wherein the edge line is a polygonal line. 

37. The method according to claim 33, wherein the edge line is a curve. 

30 

38. An image retrieval method for retrieving an image similar to an input pattern, comprising the steps of: 

calculating a feature amount of Image data in correspondence with a plurality of resolutions; 
generating search Information on the basis of the feature amount calculated in the calculating step; 
35 storing the search information generated in the generation step in units of resolutions; and 

searching for an image similar to an input pattern on the basis of the search information stored in the storing 
step. 

39. The method according to claim 38, further comprising a resolution determination step of determining a resolution 
40 used for search on the basis of vertical and horizontal dimensions and an area of each of regions in the input pattern. 

40. The method according to claim 38, wherein the calculation step includes a step of calculating the feature amount 
by calculating a cobr histogram of pixel data In units of resolutions. 

45 41. The method according to claim 40, wherein the color histogram Is a histogram of a hue in an HSV colorimetric 
system. 

42. The method according to claim 40, wherein the feature amount is calculated on the basis of the number of pixels 
within a predetermined hue overlapping range. 

so 

43. The method according to claim 33, wherein the search information includes numbers of an Image and a regbn 
having not less than a predetermined frequency of a specific color at each of the resolutions. 



44. An image retrieval method for searching a plurality of stored images for a desired Image, comprising the steps of: 

55 

inputting an image to be retrieved Including a closed region; 

comparing cobrs of regions obtained by dividing each of the stored images with a cobr of the closed region 
of the image to be retrieved, and comparing a total of areas of the regions having approximate colors of the 
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stored image with an area of the closed region; 

determining, as an image to be retrieved, a stored image in which a total of the areas of the regions is not 
considerably smaller than the area of the closed region as a result of the comparison; 
calculating a similarity by discriminating matching between the stored Image determined as the image to be 
5 retrieved, and the image to be retrieved; and 

displaying the image which has a similarity not less than a predetermined value as a retrieval result. 

45. The method according to claim 44, wherein the plurality of stored images are stored in such a manner that each 
of the plurality of stored images is divided into a plurality of regions, and sets of colors and areas of the plurality 

10 of regions are mapped on a cobr space. 

46. The method according to claim 44. wherein the image to be retrieved includes the closed region having a cobr. a 



IS 47. The method according to claim 44, wherein the retrieval results are displayed by sorting the retrieval results in a 
descending order from the highest similarity. 

48. A data carrier carrying data for causing a computer to carry out a method according to any one of claims 28 to 47. 
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